Amendments to the Claims 
Please amend Claim 22 to read as follows. 



1.-14. (Cancelled) 

15. (Previously Presented) A mixed reality space image generation method 
for generating a mixed reality space image which makes a player experience mixed reality 
by mixing a real space in which markers serving as position indices are laid out, and a 
virtual space, comprising: 

a marker detection step of detecting the markers from image data obtained 
by sensing the real space from a substantially viewpoint position of the player; 

a correcting step of correcting location/orientation information of the player 
based on the markers detected by said marker detection step; and 

a mixed reality space image generation step of generating a mixed reality 
space image to be observed by the player, so the player observes virtual object images that 
do not include any images of the markers in surrounding regions (marker regions) 
including the markers in the image data, 

wherein the mixed reality space image generation step includes a step of 
substituting or overlaying images of the marker regions by predetermined virtual object 
images. 



16. (Previously Presented) A mixed reality space image generation method 
for generating a mixed reality space image which makes a player experience mixed reality 
by mixing a real space in which markers serving as position indices are laid out, and a 
virtual space, comprising: 

a marker detection step of detecting the markers from image data obtained 
by sensing the real space from a substantially viewpoint position of the player; 

a correcting step of correcting location/orientation information of the player 
based on the markers detected by the marker detection step; and 

a mixed reality space image generation step of generating a mixed reality 
space image to be observed by the player, so the player observes virtual object images that 
do not include any images of the markers in surrounding regions (marker regions) 
including the markers from the image data, 

wherein the mixed reality space image generation step includes a step of 
substituting or overlaying an image of a region including all the markers in the image data 
by a predetermined virtual object image. 

17. (Previously Presented) The method according to claim 15, wherein the 
predetermined virtual objects are images of plane patches with the same or similar texture, 
size, location and orientation of the marker regions before the markers are laid out. 
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18. (Previously Presented) A computer readable storage medium that 
stores a program which can be executed by a computer, making the computer which 
executes the program function as: 

marker detection means for detecting the markers from image data obtained 
by sensing the real space from a substantially viewpoint position of the player; 

correcting means for correcting location/orientation information of the 
player based on the markers detected by the marker detection means; and 

mixed reality space image generation means for generating a mixed reality 
space image to be observed by the player, so the player observes virtual object images that 
do not include any images of the markers in surrounding regions (marker regions) 
including the markers in the image data, 

wherein the mixed reality space image generation means includes means for 
substituting or overlaying images of the marker regions by predetermined virtual object 
images. 

19. (Previously Presented) A computer readable storage medium that 
stores a program which can be executed by a computer, said program making the computer 
which executes the program fiinction as: 

a mixed reality apparatus for making a player experience mixed reality by 
making the player observe a mixed reality space image obtained by mixing a real space in 
which markers serving as position indices are laid out, and a virtual space, comprising: 



marker detection means for detecting the markers from image data obtained 
by sensing the real space from a substantially viewpoint position of the player; 

correcting means correcting location/orientation information of the player 
based on the markers detected by the marker detection means; and 

mixed reality space image generation means for generating a mixed reality 
space image to be observed by the player, so the player observes virtual object images that 
do not include any images of the markers in surrounding regions (marker regions) 
including the markers in the image data, 

wherein the mixed reality space image generation means includes means for 
substituting or overlaying an image of a region including all the markers in the image data 
by a predetermined virtual object image. 

20. (Previously Presented) A mixed reality apparatus for providing a mixed 
reality space shared by a plurality of players, comprising: 

an input unit adapted to input a real space image representing the real space; 

a marker detecting unit adapted to extract pixels having a color that is 
predefined as a marker from the real space image, perform a labeling process to detect 
marker regions, and detect positions of markers from the marker regions; 

a head location/orientation estimation unit to estimate a location/orientation 
of a viewpoint of a player by using the detected marker positions; 
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a mixed reality space image generation unit adapted to generate a virtual 
space image to be observed by the player based on the location/orientation of the viewpoint 
of the player and model information; and 

an image combining unit adapted to generate a mixed reality space image by 
combining the real space image with the virtual space image, 

wherein the markers are laid out in the real space so that the markers to be 
used by only the given player are laid out at positions hidden by real objects when the 
markers are observed from the other player. 

2 1 . (Previously Presented) A method for providing a mixed reality space 
shared by a plurality of players, comprising: 

an input step of inputting a real space image representing the real space; 

a marker detecting step of extracting pixels having a color that is predefined 
as a marker from the real space image, performing a labeling process to detect marker 
regions, and detecting positions of markers from the marker regions; 

a head location/orientation estimation step of estimating a 
location/orientation of a viewpoint of a player by using the detected marker positions; 

a mixed reality space image generation step of generating a virtual space 
image to be observed by the player based on the location/orientation of the viewpoint of the 
player and model information; and 

an image combining step of generating a mixed reality space image by 
combining the real space image with the virtual space image, 



wherein the markers are laid out in the real space so that the markers to be 
used by only a given player are laid out at positions hidden by real objects when the 
markers are observed from the other player. 



22. (Currently Amended) A mixed reality apparatus having an image 
generating unit adapted to generate a virtual space image corresponding to a viewpoint 
position and orientation of a player, an input unit adapted to input a real space image 
obtained by sensing from a substantially viewpoint position of the player and an image 
combining unit adapted to generate a mixed reality space image by combining the real 
space image with the virtual space image, comprising: 

a marker detection unit adapted to detect marker regions for the real space 
image input from said input unit and obtained by sensing the real space in which markers 
serving as position indices are laid out; 

a correction unit adapted to correct location/orientation information of the 
player based on the marker regions detected by said marker detection unit; 

a location/orientation estimation unit adapted t o estimate a 
location/ or i e ntation of a vi e wpoint of a given playe r using the detected mai - kcr regi o ns; 

a storing unit adapted to store a real space image obtained by sensing the 
real space before the markers are laid out as a texture image; and 

a replacing unit adapted to replace said marker regions in the real space 
image with corresponding regions in said texture image or superimpose said corresponding 
regions in said texture image on said marker regions in the real space image. 
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23. (Previously Presented) A mixed reality image generating method 
including an image generating step of generating a virtual image corresponding to a 
viewpoint position and orientation of a player, an input step of inputting a real space image 
obtained by sensing from a substantially viewpoint position of the player and an image 
combining step of generating a mixed reality space image by combining the real space 
image with the virtual space image, comprising: 

a marker detection step of detecting marker regions for the real space image 
obtained by sensing the real space in which markers serving as position indices are laid out; 

a correction step of correcting location/orientation information of the player 
based on the marker regions detected by said marker detection step; 

a storing step of storing a real space image obtained by sensing the real 
space before the markers are laid out as a texture image; and 

a replacing step of replacing said marker regions in the real space image 
with corresponding regions in said texture or superimposing said corresponding regions in 
said texture image on said marker regions in the real space image. 

24. (Previously Presented) The mixed reality generating method according 
to claim 23, wherein the same texture image is used for replace or superimpose the marker 
regions corresponding to markers laid out on real objects made of an identical material. 

25. (Previously Presented) The mixed reality image generating method 
according to claim 23 further comprising: 
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a location/orientation estimation step of estimating a location/orientation of 
a viewpoint of the player using the detected marker regions, the location information of the 
detected markers and output signal from a position/orientation sensor mounted on the 
player's head. 

26. (Previously Presented) A computer-readable storage medium that stores 
program code for controlling an apparatus to perform a mixed reality generating method, 
wherein the generating method includes an image generating step of generating a virtual 
image corresponding to a viewpoint position and orientation of a player, an input step of 
inputting a real space image obtained by sensing from substantially a viewpoint position of 
the player and an image combining step of generating a mixed reality space image by 
combining the real space image with the virtual space image, said storage medium 
comprising program code for controlling the apparatus to perform: 

a marker detection step of detecting marker regions for the real space image 
obtained by sensing the real space in which markers serving as position indices are laid out; 

a correction step of correcting location/orientation information of the player 
based on the marker regions detected by said marker detection step; 

a storing step of storing a real space image obtained by sensing the real 
space before the markers are laid out as a texture image; and 

a replacing step of replacing said marker regions in the real space image 
with corresponding regions in said texture or superimposing said corresponding regions in 
said texture image on said marker regions in the real space image. 
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